Software development kit driven features and events

ABSTRACT

A method includes providing, to a third-party content provider, code identifying a feature associated with digital content of the third-party content provider, wherein the identification comprises a feature variable placeholder associated with the feature. The method further includes receiving, by an experimentation platform, an indication of a first event type associated with the digital content, wherein the first event type identifies a first trackable action associated with the feature. The method further includes determining, by a processing device of the experimentation platform, that the first event type does not exist in a configuration associated with the feature; The method further includes, in response to the determining, generating the first event type in the configuration, generating and providing a graphical representation of the first event type for display in a user interface of the experimentation platform, and tracking the first event type on the experimentation platform.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/660,537, filed Oct. 22, 2019, the entire contents of which are herebyincorporated by reference.

FIELD OF TECHNOLOGY

The embodiments described herein pertain in general to conductingexperiments on a web page or digital product and more particularly tousing software development kit (SDK) driven features and events infeature testing on web pages or digital products.

BACKGROUND

Content variation testing is an experiment that tests changes to contentagainst a current version of the content. One example of contentvariation testing is web page variation testing where an experiment isconducted that tests a web page (or one or more elements on the page)against another design of the web page (or one or more variations of theelements on the page). During variation testing of a web page, userinteraction with a web page or its variations are measured (for example,whether a user clicks through a button or a variation of a button), andthe changes to a web page (or its elements) that produce positive ornegative results are determined. The positive results associated with aparticular change to a web page validates that the change to the webpage should be used in a production version of the web page.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a variant testing system environment,according to one embodiment.

FIGS. 2A and 2B are example variations of a web page, according to oneembodiment.

FIG. 3A is a first flow diagram of a method of using SDK-driven featuresand events in feature testing on web pages or digital products, inaccordance with some embodiments of the present disclosure.

FIG. 3B is a second flow diagram of a method of using SDK-drivenfeatures and events in feature testing on web pages or digital products,in accordance with some embodiments of the present disclosure.

FIGS. 4A-M are user interfaces for configuring an SDK, according to oneembodiment.

FIG. 5 is a block diagram of an example apparatus that may perform oneor more of the operations described herein, in accordance with someembodiments

The figures depict an embodiment for purposes of illustration only. Oneskilled in the art will readily recognize from the following descriptionthat alternative embodiments of the structures and methods illustratedherein may be employed without departing from the principles describedherein.

DETAILED DESCRIPTION

Product development teams may run A/B tests to answer specific questionsregarding which version of a web page or digital product (includingvarious features) are more successful. In some embodiments, the processof developing a product does not lend itself well to a simple A/B test.Where A/B tests have a clear beginning and end, the development of a newfeature may not. In many instances feature development may beimprecise—e.g., a development team might start by launching a barebonesprototype, then run experiments to collect data and iterate towardsperfection. Throughout the development process, various gaps in theprocess of developing, deploying, and making decisions about featuresmay exist, which may reduce the efficiency and accuracy of developmentsoftware and the computer systems on which it runs.

For example, traditional testing platforms may not provide efficientsolutions pertaining to deploying features to some users and not others,toggling features on and off, gradually rolling out features to users,A/B testing features, targeting features to specific users, remotelyconfiguring features without deploying new code, etc. Furthermore, theinteraction between development teams responsible for designingexperiments and experimentation platform program managers responsiblefor initializing experiments may cause unnecessary errors and delays.

Advantageously, the embodiments described herein provide efficientsolutions to the above, and other, problems by providing for usingSDK-driven features and events in feature testing on web pages ordigital products. As described herein, a feature is any element, orcombination of elements, of a web page or other digital product that maybe provided to a client device.

Advantageously, the embodiments described herein allow for defining anddeploying a feature and/or event once, by a developer via an SDK insteadof by a program manager in a user interface (UI). A representation ofthe feature and/or event created via the SDK may then be added to theUI. The UI may then be used to toggle textures and events on or offremotely, run experiments to collect data to understand whether a newfeature works (e.g., via AB tests on features and gradual rollouts offeatures to users), and effectively manage features to ensure they areworking as expected. It should be noted that server-side configurationvariables are also referred to herein as feature variables, merely forconvenience and brevity.

FIG. 1 is a block diagram of a variant testing system environment 100according to one embodiment. Environment 100 includes client devices 102(e.g., client device 102A and client device 102B). In one embodiment,client device 102A is a device of a first user and client device 102B isa device of a second user. Client device 102A may represent one or moredevices of the first user and client device 102B may represent one ormore devices of the second user. Client devices 102 are connected to anexperimentation system 105 and a third-party content provider 107 via anetwork 109. Although the environment 100 shown in FIG. 1 only includestwo client devices 102, the environment 100 can include any number ofclient devices (e.g., thousands of client devices 102).

FIG. 1 and the other figures may use like reference numerals to identifylike elements. A letter after a reference numeral, such as “102A,”indicates that the text refers specifically to the element having thatparticular reference numeral. A reference numeral in the text without afollowing letter, such as “102,” refers to any or all of the elements inthe figures bearing that reference numeral.

Network 109 enables communication among the entities connected to it. Inone embodiment, network 109 is the Internet and uses standardcommunications technologies and/or protocols. Thus, network 109 caninclude links using technologies such as Ethernet, WiFi (e.g., 802.11),worldwide interoperability for microwave access (WiMAX), 3G, Long TermEvolution (LTE), digital subscriber line (DSL), asynchronous transfermode (ATM), InfiniBand, etc. Similarly, the networking protocols used onnetwork 109 can include multiprotocol label switching (MPLS), thetransmission control protocol/Internet protocol (TCP/IP), the UserDatagram Protocol (UDP), the hypertext transport protocol (HTTP), thesimple mail transfer protocol (SMTP), the file transfer protocol (FTP),etc. The data exchanged over the network 109 can be represented usingtechnologies and/or formats including the hypertext markup language(HTML), the extensible markup language (XML), etc. In addition, all orsome of links can be encrypted using conventional encryptiontechnologies such as the secure sockets layer (SSL), transport layersecurity (TLS), virtual private networks (VPNs), Internet Protocolsecurity (IPsec), etc. In one embodiment, network 109 may include but isnot limited to any combination of a local area network (LAN), ametropolitan area network (MAN), and a wide area network (WAN). Inanother embodiment, the entities use custom and/or dedicated datacommunications technologies instead of, or in addition to, the onesdescribed above.

Client devices 102 may communicate with the third-party content provider107 via the network 109 to receive content items such as web pages fromthe third-party content provider 107. In one embodiment, each clientdevice 102 is a computer system capable of communicating with theexperimentation system 105 and the third-party content provider 107.Examples of client devices 102 include a personal computer, a mobilephone, a tablet, a personal digital assistant (PDA), or a laptop. Asshown in FIG. 1 , each client device 102 includes a web browser 103(e.g., web browser 103A and web browser 103B). Web browser 103 is acomputer program stored on a client device 102 that allows the user ofclient 102 to access web pages on the World Wide Web such as web pagesprovided by the third-party content provider 107.

As mentioned above, the environment 100 includes a third-party contentprovider 107. Hereinafter, the third-party content provider 107 isreferred to as a “content provider 107” for ease of description. Thecontent provider 107 may be an individual, a group of individuals, or anentity such as a company. In one embodiment, the content provider 107provides content items to client devices 102. While only one contentprovider 107 is shown, it is understood that any number of contentproviders are supported and can be in the environment 100 at any time.

In one embodiment, the content items provided by the content provider107 include web pages. However, the content provider 107 may provideother types of content items (e.g., digital content) such as softwareapplications, including web applications, mobile device applications,etc.). The web pages and software applications may include video, audio,a combination of video and audio, still images (e.g., JPEG), textdocuments, advertisements, and/or any other types of content. Forpurposes of convenience and the description of one embodiment, thecontent items provided by the content provider 107 will be referred toas a web pages, but no limitation on the type of content items areintended by this terminology.

In one embodiment, the content provider 107 operates in conjunction withthe experimentation system 105 to perform variation testing on webpages. The content provider 107 may display different variations of aweb page, including different features, to client devices 102 based oninstructions from the experimentation system 105 that is conducting avariation test on the web page for the content provider 107. A variationtest for a web page may test changes to the web page against the currentvariation of the web page to determine how the variation alters userinteraction, or some other metric, with the web page or whether thechanges produce a desired result. An example of a desired resultresulting from a change to a web page is an increased selection of anadvertisement(s) included in the web page or increased purchases of aproduct advertised on a web page. Thus, variation testing validates anew design of a web page or changes on elements on the web page beforethe new design or changes are put into production by the contentprovider 107.

For a given web page, the content provider 107 may have one or morevariations of the web page that are used in a variation test for the webpage. In one embodiment, a variation test of a web page involves anexperiment that tests an “A” variation known as the control and a “B”variation known as the variant on users requesting the web page from thecontent provider 107. For ease of discussion, the embodiments discussedherein describe a web page having only two variations: the control andthe variant. However, in other embodiments, a web page can have anynumber of variants.

FIG. 2A is an example of an “A” variation (e.g., the control variation)of a web page 200A that represents the current implementation of the webpage provided by content provider 107. The control variation of the webpage 200A includes a website name of the website associated with the webpage, a first picture, and a first advertisement. The website name, thefirst picture, and the first advertisement are located in an upperportion 201 of the control variation of web page 200A according to oneembodiment. The control variation of web page 200A also includes textualcontent located in a central portion 203 of the control variation of webpage 200A and the control variation of the web page 200A also includes asecond picture and a second advertisement located at a lower portion 205of the control variation of the web page 200A.

FIG. 2B is an example of a “B” variation (e.g., the variant version) ofa web page e 200B. The variant version of the web page represented byweb page 200B includes a change (i.e., a modification) to the controlversion of the web page 200A shown in FIG. 2A. The variant version ofweb page 200B includes the website name, a first advertisement, a secondadvertisement, textual content, a first picture, and a second picturesimilar to the control version of the web page 200A shown in FIG. 2A.However, the variant version of web page 200B includes the secondadvertisement positioned in the upper portion 201 of web page 200Bwhereas the second advertisement is positioned in the lower portion 205in the control version of the web page 200A. In one embodiment, thevariation test using the control version and the variant version of theweb page is conducted to determine whether the second advertisementreceives more selections when the second advertisement is located at thelower portion 205 of the web page as shown in FIG. 2A or when the secondadvertisement is located at the upper portion 201 of the web page asshown in FIG. 2B.

Referring back to FIG. 1 , in one embodiment the content provider 107includes a web server 111, web page 119, and web page content 123. Theexperimentation system 105 includes experiment setup logic 113, contentdelivery logic 118, identification (ID) assignment logic 115, an IDdatabase 116, and an experiment database 117. As is known in the art,the term “logic” refers to computer program logic utilized to providethe specified functionality. Thus, logic can be implemented in hardware,firmware, and/or software. In one embodiment, program logic is stored ona non-transitory storage device (i.e., a computer program product),loaded into a memory, and executed by one or more computer processors.Additionally, those of skill in the art will recognize that otherembodiments of the content provider 107 shown in FIG. 1 can havedifferent and/or other logic than that described here, and that thefunctionalities can be distributed among the logic in a differentmanner.

In one embodiment, the experiment database 117 stores multiple (i.e., aplurality) variation testing records that describe variation testsconducted by the experimentation system 105. Each variation test recordis associated with a web page (e.g., whether a static web page or a webpage), an experiment identifier unique to the variation test, andidentifiers of different variations (e.g., having different features) ofa web page involved in the variation test. A variation test record mayalso include a description of the change(s) made to each variation ofthe web page compared to the control variation of the web page.

In one embodiment, the web server 109 links the content provider 107 toone or more client devices 102 and the experimentation system 105 viathe network 109. The web server 111 serves content items such as webpages, as well as other web related content, such as Java, Flash, XML,and so forth. The web server 111 may include a mail server or othermessaging functionality for receiving and routing messages between theexperimentation system 105 and one or more client devices 102. In oneembodiment, web server 111 includes configuration 124, which may be adata file or any other suitable storage structure. In one embodiment,various experiment settings are represented in JSON within configuration111. In another embodiment, any other suitable programming language maybe used.

In one embodiment, the identification (ID) assignment logic 115generates user identifiers for users of client devices 102. In oneembodiment, the user identifier is used by the user of a client device102 to log into a web page or application provided by the contentprovider 107. In one embodiment, a user identifier is a string ofnumbers that is unique to a particular client device 102. Thus, clientdevice 102A may have a different user identifier than the useridentifier for client device 102B. In one embodiment, the ID assignmentlogic 115 generates a user identifier by automatically incrementing thevalue of the last user identifier generated by the ID assignment logic115 using an auto increment function. Alternatively, the ID assignmentlogic 115 generates a unique random number and assigns the unique randomnumber as a user identifier for a client device 102. In anotherembodiment, the ID assignment logic 115 may receive a user identifierfrom a client device 102. The user identifier received from the clientdevice 102 may be generated by the client device 102 or by the clientloaded on the web page of the client device, or the user identifier maybe input by the user into the client device 102. The ID assignment logic115 may store user identifiers in the ID database 116 that maintains arecord of all assigned user identifiers.

In one embodiment, responsive to the content provider 107 receiving arequest for a web page from a client device, the ID assignment logic 115communicates with the client device 102 to determine whether the clientdevice 102 already has a user identifier. For example, the ID assignmentlogic 115 may query the client device 102 to transmit the useridentifier via a login form displayed on the web page. In anotherexample, the ID assignment logic 115 queries the client device 102 todetermine whether the client device 102 includes a stored cookie thatcomprises a user identifier for the client device 102. If the clientdevice 102 lacks a cookie with a user identifier for the client device102, the ID assignment logic 115 may query the client device 102 totransmit the user identifier via a login form displayed on the web page.Alternatively, the ID assignment logic 115 generates a user identifierfor the client device 102 and communicates the user identifier to theclient device 102 for storage in a cookie at the client device 102 ifthe client device 102 lacks a cookie with a user identifier.Alternatively, the client loaded on the webpage generates a cookiecontaining a randomly generated user identifier and sends this useridentifier to assignment logic 115. The client device 102 may continueto use the user identifier generated by the ID assignment logic 115 forsubsequent requests for the web page.

FIG. 3A is a first flow diagram of a method of using server-sideconfiguration variables in feature testing on web pages or digitalproducts, in accordance with some embodiments of the present disclosure.Method 300A may be performed by processing logic that may comprisehardware (e.g., circuitry, dedicated logic, programmable logic, aprocessor, a processing device, a central processing unit (CPU), asystem-on-chip (SoC), etc.), software (e.g., instructionsrunning/executing on a processing device), firmware (e.g., microcode),or a combination thereof. In some embodiments, the method 300A may beperformed by system 100 of FIG. 1 .

The method 300A begins at block 302, in which processing logic provides,to a third-party content provider, code identifying a feature associatedwith digital content of the third-party content provider. In oneembodiment, the identification includes a feature variable placeholder(e.g., a default value) associated with the feature. Optionally, theindication of the type identifies a name and type of the feature.

In one embodiment, references to the feature are included in the code asif the feature already exists on the experimentation platform, which mayor may not be accurate. Advantageously, the experimentation platform mayautomatically (e.g., without human interaction) generate the feature andother components of an experiment if they do not already exist on theplatform, as described herein. Similarly, tracking calls may be includedin the code, regardless of whether they already exist on the platform(and automatically generated if they do not). In one embodiment,tracking calls communicate what happens to the experimentation platform.Tracking calls may include the name of the event and the ID of the userthat triggered event. Tracking calls can optionally contain additionalmetadata, tags, and numerical values such as revenue.

Optionally, in response to receiving an indication that the code wasdeployed, processing logic may generate (e.g., automatically or viahuman interaction) and execute an experiment that includes the feature.At block 304, processing logic receives (e.g., by an experimentationplatform) an indication of a first event type associated with thedigital content. In one embodiment, the first event type identifies afirst trackable action associated with the feature. Events may include,for example, clicks, page views, form submissions, purchases, and anyother suitably-trackable action. In one embodiment, event data may beused to measure and select a desired variation of an experiment/feature.

In one embodiment, processing logic may optionally authenticate a userand/or request before performing operations that affect theexperimentation platform and/or user code. For example, in oneembodiment, processing logic may authenticate the code before receivingthe indication of the first event type. In one embodiment,authentication include determining that a user or user device providingcode for a web site or other digital product is privileged to access theweb site or digital product. Such authentication may include the use ofcredentials, tokens, encryption/decryption keys, etc.

At block 306, processing logic determines, by a processing device of theexperimentation platform, that the first event type does not exist in aconfiguration associated with the feature. In one embodiment, inresponse determining that the first event type does not exist,processing logic may generate (e.g., create and store) the first eventtype in the configuration 308. In one embodiment, data in theconfiguration is represented in JavaScript Object Notation (JSON). Inanother embodiment, eXtensible Markup Language (XML) or any othersuitable programming language may be used.

In another embodiment, processing logic may perform alternate operationsif it is determined that an event already exists within theconfiguration. For example, processing logic may receive an indicationof a second event type associated with the digital content, wherein thesecond event type identifies a second trackable action associated withthe feature. Processing logic may then determine that the second eventtype does exist in the configuration associated with the feature. Inresponse to the determining that the second event type does exist in theconfiguration, processing logic may not add the second event type to theconfiguration (e.g., since it already exists) and instead proceeddirectly to tracking the second event type on the experimentationplatform.

At block 310, processing logic generates and provides a graphicalrepresentation of the first event type for display in a user interfaceof the experimentation platform. In one embodiment, the graphicalelement is actionable, allowing a user to activate and deactivate thefirst event type at will.

At block 312, processing logic tracks the first event type on theexperimentation platform. To track the first event type, theexperimentation platform may continue to receive and log instances inwhich the event is activated. Optionally, processing logic may select aversion of the feature to implement based on the tracking of the firstevent type on the experimentation platform. For example, if version A ofthe feature performs better (e.g., receives more clicks, in one example)than version B, version A may be selected and provided to client devicesinstead of version B.

In one embodiment, processing logic may optionally prevent new features,events, etc. from being generated at a rate that might impactperformance of the web site or digital product. For example, in oneembodiment, processing logic may determine that the experimentationplatform is to generate a feature (e.g., a new feature is referenced incode that does not already exist in the respective configuration).Processing logic may then determine that a number of features generatedexceeds a threshold. For example, a rule may exist that allows ten newfeature creations in a ten-minute sliding window. Processing logic maydetermine that generating the new feature will cause an eleventh featureto be generated in the last ten minutes. In response to determining thatthe number of features generated exceeds the threshold (e.g., within thesliding window), processing logic may prevent the experimentationplatform from generating the feature. Optionally, processing logic maylog an error describing the exception for debugging purposes.

FIG. 3B is a first flow diagram of a method of using server-sideconfiguration variables in feature testing on web pages or digitalproducts, in accordance with some embodiments of the present disclosure.Method 300B may be performed by processing logic that may comprisehardware (e.g., circuitry, dedicated logic, programmable logic, aprocessor, a processing device, a central processing unit (CPU), asystem-on-chip (SoC), etc.), software (e.g., instructionsrunning/executing on a processing device), firmware (e.g., microcode),or a combination thereof. In some embodiments, the method 300B may beperformed by system 100 of FIG. 1 .

The method 300B begins at block 303, in which processing logicdetermines that the feature (e.g., of method 300B) does not exist in aconfiguration associated with the experiment. In one embodiment, todetermine that the feature does not exist in the configuration,processing logic at block 305 optionally may determine that the featuredoes not exist in a cache associated with the configuration. In oneembodiment, the cache is maintained by the experimentation platform toincrease the efficiency and speed of the system.

In response to determining that the feature does not exist in theconfiguration, processing logic at block 307 generates (e.g., in JSON)the feature in the configuration based on the name, type, and featurevariable placeholder (e.g., default value) of the feature. In oneembodiment, to generate the feature in the configuration, processinglogic at block 309 optionally makes use of a representational statetransfer (REST) application program interface (API) to create a requestto generate the feature on the experimentation platform. At block 311,processing logic generates a configurable representation of the featurein a user interface of the experimentation platform (e.g., as shown inFIGS. 4A-M).

FIGS. 4A-M are user interfaces for configuring an SDK, according to oneembodiment. Advantageously, the elements of the graphical userinterfaces illustrated in FIGS. 3A-M may be automatically (e.g., withouthuman interaction) generated to represent data received via the SDKdescribed herein.

In one embodiment, user interface 400 allows users to view and configurefeature flags, which were added via the SDK, as described herein. Asshown, when a new feature is generated by the SDK, a representation 402of the feature is generated and displayed on the user interface 400. Anelement 404 of the user interface 400 may allow a user to activate anddeactivate the feature 402. In one embodiment, clicking on the feature402 in the user interface 400 causes a configuration user interface 403of FIG. 4B to be displayed on a client device connecting to theexperimentation system. GUI 403 may illustrate a feature configurationform, which may allow a user to modify a unique feature key 404. In oneembodiment, feature key 404 is a unique identifier (ID) that anengineer, or a computer system, may use to determine whether aparticular feature is on or off when a program runs. In one embodiment,the feature key was generating by the SDK in response to determiningthat the feature did not already exist.

In one embodiment, GUI 403 may allow users to provide or modify anoptional description 405 to provide context about the purpose of thefeature. The feature form (e.g., 403) may include the ability to definea feature rollout 406. In one embodiment, a feature rollout may be theability to turn a feature on and to specify the percentage of traffic(e.g., network traffic) for which a feature is enabled. In oneembodiment, new feature rollouts may default to “off” with 0% of trafficallocated. In one embodiment, as shown in FIG. 4B, to turn on (e.g.,activate) a feature, a user may set the feature toggle 408 to “on” andset a traffic allocation 410. If a rollout is toggled on for a trafficpercentage, then toggled off, then later toggled back on, the rolloutmay retain the traffic percentage previously defined.

In one embodiment, to determine rollout eligibility, a system maycombine a user ID with a unique rollout ID and compute a deterministichash that outputs a number between 0 and 10,000, for example (any othernumber may be used). If that number falls within the range specified bythe feature rollout's traffic allocation, the visitor may eligible for agiven feature. In another embodiment, rollouts may also include audiencetargeting, which may be evaluated before the rollout traffic allocation.And, if there is a feature test running at the same time as the featurerollout, the feature test takes may take precedence over the rollout.

In one embodiment, users may be able to specify whether a rollout istoggled on or off and are be able to specify traffic allocation at theenvironment level. This may provide support for a variety of use cases,such as setting a feature to 100% of traffic in a pre-productionenvironment while the feature is only exposed to a small percentage ofproduction traffic. In one embodiment, rollouts may be targeted tospecific audiences.

In one embodiment, a feature dashboard 411 of FIG. 4C provides asnapshot of what is happening to a feature at a given point in time toclearly communicate what is happening and to facilitate easy paths toaction. The feature dashboard 411 may include the following columns, inone example:

-   -   Active Experiments—Lists all active (running or paused) Feature        Tests and/or MVTs on each Feature; if there are >2 active        experiments, we'll follow the pattern that we use for        Environments—show the most recent and the collapse the others as        “Other Experiments (N)”    -   Rollout—If there is an active Rollout running in the Production        Environment, this shows the traffic allocation of that Rollout;        if the Feature is off in production, this reads “Off”    -   Environment—Same component that is used on the Experiments        Dashboard; shows Production Environment status and then other        Environments rolled up    -   Created—Date Feature was created    -   Modified—Date Feature/Rollout was last modified

In one embodiment, users can take action directly from the featuredashboard 411 by clicking the ellipses 412 on the right hand side. Doingso may open a menu that includes environment-specific toggles, which maybe analogous to environment-specific status buttons available on theexperiments tab. In one embodiment, changing a toggle from the featuredashboard 411 may trigger a popup prompting the user to confirm anychanges.

In one embodiment, GUI 413 of FIG. 4D provides the option to define afeature configuration, which may be a set of one or more featurevariables (e.g., server-side configuration variables) that users can useto make a feature more powerful. In one embodiment, featureconfiguration allows users to change the values of the variables (e.g.,feature variable values) they define without deploying code. In oneembodiment, feature variables are generated and populated into GUI 413via the SDK. Once generated and populated, features variables may bemodified, added, and deleted within GUI 413.

In one embodiment, the default feature variable values defined on thefeature form are returned by feature variable accessors unless they areoverridden by a variation in an experiment. In one embodiment, if thereis a running feature test on a feature that uses a featureconfiguration, the feature configuration is locked until the test ispaused. In various embodiments, one or more of a variable key (e.g., ID)414, variable type (e.g., Boolean, String, Integer, Double, etc.) 415,or a default value 416 may be generated via the SDK, and modified viaGUI 413.

In one embodiment, the default values 416 for feature variables may bereturned under the following example circumstances:

-   -   If a feature is disabled (as a safeguard to protect customers        who are reference the variable even when the Feature is        disabled)    -   As part of a feature rollout    -   As part of a feature test, unless the default values are        overridden by a variation        In another embodiment, the system may allow users to specify        “rollout rules” and to vary feature variable values as part of        those roles.

In one embodiment, feature flags may be generated via the SDK, andgenerated and populated into GUIs where they may be added to, modified,deleted, etc. A feature flag may be a rule to enable, disable, hide, orroll out a product feature at runtime. One benefit of a feature flag isit allows a user to configure the behavior of a product on the flywithout deploying code. For example, if a company deploys a new featurebehind a feature flag, they can subsequently enable it for a small groupof their users and hide it for everyone else.

In one embodiment, existing feature flags (e.g., generated via an SDK)may be displayed in GUI 417 of FIG. 4E. In one embodiment, feature flagscan use audiences for targeting, just like experiments. For example, itis possible to map an ordered set of audiences to different feature flagvalues, i.e. if the user is in audience A1 they should get value V1,otherwise if they are in audience A2 they should get value V2, and soon. The GUI for such may look like GUI 419 of FIG. 4F.

For example, if a user is in an Enterprise A account 420, the featureshould be disabled 421. Otherwise, if a user is an OTT Beta Customer422, the feature should be enabled 423, etc. For other flag types (i.e.Boolean, Integer, Number, Strings, etc.) a free form text box may beprovided to enter feature values. Features may be further enabled anddisabled for subsets of accounts, as shown in GUI 424 of FIG. 4G. In oneembodiment, features may be rolled out according to the followingexample logic: If the user is eligible for an experiment using featureF, get the feature value from the experiment; If the user is noteligible for an experiment, get the feature value from ordered audiencesfor feature F; Otherwise, use the default value for feature F.

In one embodiment, as shown in GUI 425 of FIG. 4H, rollouts may havemetrics describing whether a feature was used for a given customer, asthat information will be available from raw data of feature access.Rollouts may include a blacklist audience listing all the IDs of peoplewho should NOT have access to the feature. In one embodiment, there maybe a special “audience” that sits atop all other targeting rules andhelps enforce whitelisting behavior. In one embodiment, whitelists 426(and blacklists) may be explicitly called out in the GUI. In oneembodiment, for whitelists specifically, an Uploaded Audiences featuremay be included that is essentially the same as the whitelist.

In one embodiment, rollouts may have a 1:1 relationship with a feature.They represent a set of targeting rules for a feature which are appliedin order. To the end user, they will be defining audiences and enablingthem for a feature. The system may create a campaign (e.g., model namelayer) and create experiences (e.g., model name layer experiment) equalto the number of audiences with each experience being represented by theaudience. When applying these rules to a rollout, a “variation”belonging to a layer experiment (e.g., corresponding to an audience) mayresult and the values associated with an audience may be stored in afeature variables model.

In one embodiment, a rollout may have a special rule called an “EveryoneElse” rule. While evaluating audiences, if the user meets the audienceconditions, but does not get the feature because of the trafficallocation, then all intermediate rules may be skipped and the systemwill directly evaluate the “Everyone Else” rule.

In one embodiment, GUI 427 of FIG. 4I allows customers to upload largelists of values 428 that can be used to create attribute conditions 429in the audience builder GUI 425 (e.g., account ID can be any of thefollowing 500 values . . . ). In one embodiment, a user ID may be passedinto targeting logic as an attribute, so users can effectively create anaudience used as a whitelist.

In one embodiment, GUI 430 of FIG. 4J includes a variety of experiments,including AB test, feature test, and/or multivariate test. To create afeature test, a user may select “Create New Experiment” element 431 andchoose “Feature Test” 432 from the dropdown. In one embodiment, thiswill open a new form (GUI 433 of FIG. 4K).

In GUI 433 of FIG. 4K, users may use a searchable text field 434 tochoose an existing feature (e.g., one added via the SDK), or can createa new feature 435 before creating a test. Once the user selects afeature (or creates a new one), the system may automatically generate anexperiment key 436 by appending “test” to the end of the feature key. Inone embodiment, this experiment key is editable. In the event there isalready a feature test that uses the automatic experiment key, thesystem may will append “_test1,” “_test2\,” etc.

In one embodiment, feature tests may automatically suggest variationkeys. By default, the system may provide two variations with keys“variation_1” and “variation_2.” Like experiment keys, variation keysmay be editable. If the user adds additional variations, the system mayprovide automatic suggestions according to the variation number:“variation_3,” “variation_4,” etc. If the user deletes a variation thesystem may continue incrementing as though the deleted variation stillexists.

In one embodiment, feature test variations may expose a feature toggle438 and the feature's configuration 439 (if one exists), as shown in GUI437 of FIG. 4L. By default, the toggle 438 may be set to “on” and theconfiguration 439 will load its default value(s) 440 (e.g., as providedvia the SDK). In one embodiment, a feature may not include aconfiguration, so a future test may be run by testing one variationconsisting of “Toggle=on” and another variation consisting of“Toggle=off.” If a feature includes a feature configuration, and if theuser sets a variation to “Toggle=off,” the system may disable the optionto modify variable values and may revert to the default variable values.

Advantageously, the embodiments provided herein allow for the ability torun experiments on features that include feature configurations—usersare be able to initialize their applications with features and featureconfigurations, then run unlimited numbers of experiments without havingto perform code redeploys. To create variations using featureconfigurations, a user may update the variable values (e.g., 440, 441)under each variation on the GUI 437. In one embodiment once theexperiment is live, an application programming interface (API) mayreturn the values specified for the variation assigned to a user.

In one embodiment, if there exist feature tests for a feature, thefeature form (e.g., GUI 442 of FIG. 4M) may provide information aboutthose tests. For example, for an experiment running on a feature GUI 442may provide one or more of:

-   -   An Experiment Key 443: unique identifier for the experiment;        links to the Experiment Form    -   A Type 444: to start, this will always be “Feature Test”, but        will include MVT and potentially other types in the future    -   A Status 445: Draft/Running/Paused

In one embodiment, feature tests take precedence over feature rollouts.In other words, if both a feature test and feature rollout are active,the feature test is evaluated first. If a user does not qualify for thefeature test, the feature rollout is evaluated. To illustrate how thismight work in practice, the following hypothetical describes possibleoutcomes:

Scenario:

Feature Test running with an Audience and a Traffic Allocation

Feature Rollout running with an Audience and a Traffic Allocation

Acronyms

FTA=Feature Test Audience

FTT=Feature Test Traffic Allocation

FRA=Feature Rollout Audience

FRT=Feature Rollout Traffic Allocation

User FTA FTT FRA FTT Result user1 passes passes n/a n/a Feature Testuser2 passes fails passes passes Feature Rollout user3 fails n/a passespasses Feature Rollout user4 fails n/a passes fails No action user5fails n/a fails n/a No action

In one embodiment, users may be allowed a maximum of one runningproduction feature test at a time, unless the feature is assigned to amutually exclusive group. Users may be allowed multiple draft, paused,and archived experiments on a feature. If a user tries to launch asecond feature test, on while another feature test is running, thesystem may warn the user that a maximum of one feature test is allowedat a time (unless the experiments are mutex). In one embodiment,mutually exclusive groups may allow users to run concurrent productionfeature tests. In the event that there are feature tests running in amutex group, evaluating the users mutex group becomes the first step inthe order of operations. Once the mutex group is assigned, the systemmay evaluate feature tests and rollouts according to the same rulesdescribed above.

In one embodiment, anyone may create a feature. Likewise, anyone maycreate and enable feature rollouts and feature tests in non-productionenvironments. IN another embodiment, only Admins, Project Owners, andPublishers can enable feature rollouts and feature tests in production.

In one embodiment, feature toggles may be added to feature rollouts andmake toggles compatible with environments:

Feature Dashboard

-   -   Environment-specific Feature Toggles

Feature Form

-   -   Add Environment-specific Feature Toggles        Other features of the system described herein may include:    -   Cleaning up Features: Provide the ability to track when the code        used to launch a feature should be removed    -   Rollout Results: Provide results specifically designed to        address the “gradual rollout to mitigate risk” use case. These        may deviate from traditional AB test results. For example,        rollout results may measure sample populations that change as        traffic allocations change, and would be designed to measure        “not worse” by default, rather than “is better.”    -   Advanced Targeting, Whitelisting & Blacklisting: Improve        audience targeting to allow customers to define types of        audiences that aren't well support today. For example, add the        ability to match attributes using operators like “list includes”        or “matches regex”; add the ability to provide typed attributes        like integers or dates; add the ability to specify lists of user        IDs to whitelist or blacklist for a rollout or feature test    -   Advanced Rollout Rules: Provide the ability to specify different        feature configurations for different audiences.

In one embodiment, the third-party content provider may connect awebsite (or other digital content) to the experimentation platformusing, e.g., an API. This connection may occur locally or via a network.The connection permits the experimentation platform and websiteserver(s) to communicate. In one embodiment, code on the website may beconfigured to check the feature flags that have been configured on theexperimentation platform before loading a web page on the client device.The experimentation platform may use the connection to control the logicthat determines whether or not to deploy a feature on the web page. Insome embodiments, the experimentation platform may also control whichfeatures variables to deploy (e.g., depending on a scenario or a groupof users). In various embodiments, features, feature variables, and/orfeature variable values are created and sent from a third-party contentcreator, and received by processing logic of the systems describedherein (e.g., via the connection established between the experimentationplatform and the website server(s)).

As an illustrative example of the setup process described herein, afterconnecting its website to the experimentation system as discussed above,a content provider may create code for a feature, such as a new sign-inform for the website (the “feature”). The code for the feature includesa unique feature key that permits that feature to be identified. Thecode includes a placeholder variable that serves a placeholder for textfor the sign-in banner (e.g. “Sign in here”). The experimentationsystem, using the user interface described in FIG. 4A, may allowconfiguration of a feature flag associated with the feature. Theconfiguration includes the unique feature key that matches the featurekey for the feature on the website. The content provider configures arule on the experimentation system to determine when and to whom the newsign-in feature should be deployed (e.g., use the new form with 40% ofwebsite visitors, else use old sign-in form). The content provider canalso set up variables (e.g., feature variables) on the experimentationsystems with different variations of the sign-in banner text (e.g.“Welcome San Francisco people” and “Welcome! Thank you for usingFirefox”). Those variables may be associated with audiences who willreceive the variation (e.g., people in San Francisco and people who useFirefox). Variables can also be triggered based on other scenarios (e.g.time-of-day, etc.).

When a client device connects to the content provider's website (orother digital content), the client device makes a request for a web page(or other digital content) that may optionally include the new sign-inform. The code on the web site checks with the experimentation platformto determine whether the client device should receive the new sign-inform. Using the configurations discussed above, the experimentationplatform directs the web site whether to deploy the new sign in form tothe client device. In further embodiments, the experimentation platformdetermines which of the different variations of the sign-in banner textto deploy to the particular client device. Once the determination ismade, the experimentation platform sends the applicable sign-in bannertext to the web site, which inserts the text in place of the featurevariable discussed above. The web server may then send the web page withthe new feature and feature variable variation (e.g., value) to theclient device.

In one embodiment, the selection of the sign-in banner text (e.g., thefeature variable value) is made randomly. In other embodiments, theselection and depends on the characteristics of the client device, suchas geographic location of the client device. In still other embodiments,the selection is made based on data gathered about the user of theclient device.

In further embodiments, the web site sends data to the experimentationplatform regarding whether a client device who sees the sign in bannertext variation actually proceeds to log into the website. Theexperimentation platform can aggregate this data and determine which ofthe banner text variations leads to more log ins to the website.

In one embodiment, the feature variable value consists of a text string.In other embodiments, the feature variable value can consist of Booleanvalues, integers, double, data objects, images, videos, and othercontent.

One advantage of the above is that the content provider does not have tochange the actual code of the feature (e.g., go into their code base tochange the banner text from default text to the San Francisco User text,as described in the above example). The content provided can instead dothis by setting up variables in the experimentation system. Worthnoting, banner text is one example of a feature variable, but there areseveral other applications.

FIG. 5 is a block diagram of an example computing device 500 that mayperform one or more of the operations described herein, in accordancewith some embodiments. In various embodiments, computing device 500 mayrepresent computing devices (e.g., servers) of the experimentationplatform, third-party content provider client devices, and/orthird-party content provider servers. Computing device 500 may beconnected to other computing devices in a LAN, an intranet, an extranet,and/or the Internet. The computing device may operate in the capacity ofa server machine in client-server network environment or in the capacityof a client in a peer-to-peer network environment. The computing devicemay be provided by a personal computer (PC), a set-top box (STB), aserver, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlecomputing device is illustrated, the term “computing device” shall alsobe taken to include any collection of computing devices thatindividually or jointly execute a set (or multiple sets) of instructionsto perform the methods discussed herein.

The example computing device 500 may include a processing device (e.g.,a general purpose processor, a PLD, etc.) 502, a main memory 504 (e.g.,synchronous dynamic random access memory (DRAM), read-only memory(ROM)), a static memory 506 (e.g., flash memory and a data storagedevice 518), which may communicate with each other via a bus 530.

Processing device 502 may be provided by one or more general-purposeprocessing devices such as a microprocessor, central processing unit, orthe like. In an illustrative example, processing device 502 may comprisea complex instruction set computing (CISC) microprocessor, reducedinstruction set computing (RISC) microprocessor, very long instructionword (VLIW) microprocessor, or a processor implementing otherinstruction sets or processors implementing a combination of instructionsets. Processing device 502 may also comprise one or morespecial-purpose processing devices such as an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), network processor, or the like. Theprocessing device 502 may be configured to execute the operationsdescribed herein, in accordance with one or more aspects of the presentdisclosure, for performing the operations and steps discussed herein.

Computing device 500 may further include a network interface device 508which may communicate with a network 520. The computing device 500 alsomay include a video display unit 510 (e.g., a liquid crystal display(LCD) or a cathode ray tube (CRT)), an alphanumeric input device 512(e.g., a keyboard), a cursor control device 514 (e.g., a mouse) and anacoustic signal generation device 516 (e.g., a speaker). In oneembodiment, video display unit 510, alphanumeric input device 512, andcursor control device 514 may be combined into a single component ordevice (e.g., an LCD touch screen).

Data storage device 518 may include a computer-readable storage medium528 on which may be stored one or more sets of instructions, e.g.,instructions for carrying out the operations described herein, inaccordance with one or more aspects of the present disclosure. SDKinstructions 526 may also reside, completely or at least partially,within main memory 504 and/or within processing device 502 duringexecution thereof by computing device 500, main memory 504 andprocessing device 502 also constituting computer-readable media. Theinstructions may further be transmitted or received over a network 520via network interface device 508.

While computer-readable storage medium 528 is shown in an illustrativeexample to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing, encoding or carrying a set ofinstructions for execution by the machine and that cause the machine toperform the methods described herein. The term “computer-readablestorage medium” shall accordingly be taken to include, but not belimited to, solid-state memories, optical media and magnetic media.

The methods and illustrative examples described herein are notinherently related to any particular computer or other apparatus.Various general purpose systems may be used in accordance with theteachings described herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will appear as set forth in thedescription above.

The above description is intended to be illustrative, and notrestrictive. Although the present disclosure has been described withreferences to specific illustrative examples, it will be recognized thatthe present disclosure is not limited to the examples described. Thescope of the disclosure should be determined with reference to thefollowing claims, along with the full scope of equivalents to which theclaims are entitled.

As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising”, “includes”, and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Therefore, the terminology usedherein is for the purpose of describing particular embodiments only andis not intended to be limiting.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Although the method operations were described in a specific order, itshould be understood that other operations may be performed in betweendescribed operations, described operations may be adjusted so that theyoccur at slightly different times or the described operations may bedistributed in a system which allows the occurrence of the processingoperations at various intervals associated with the processing.

Various units, circuits, or other components may be described or claimedas “configured to” or “configurable to” perform a task or tasks. In suchcontexts, the phrase “configured to” or “configurable to” is used toconnote structure by indicating that the units/circuits/componentsinclude structure (e.g., circuitry) that performs the task or tasksduring operation. As such, the unit/circuit/component can be said to beconfigured to perform the task, or configurable to perform the task,even when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” or “configurable to” language include hardware—forexample, circuits, memory storing program instructions executable toimplement the operation, etc. Reciting that a unit/circuit/component is“configured to” perform one or more tasks, or is “configurable to”perform one or more tasks, is expressly intended not to invoke 35 U.S.C.112, sixth paragraph, for that unit/circuit/component. Additionally,“configured to” or “configurable to” can include generic structure(e.g., generic circuitry) that is manipulated by software and/orfirmware (e.g., an FPGA or a general-purpose processor executingsoftware) to operate in manner that is capable of performing the task(s)at issue. “Configured to” may also include adapting a manufacturingprocess (e.g., a semiconductor fabrication facility) to fabricatedevices (e.g., integrated circuits) that are adapted to implement orperform one or more tasks. “Configurable to” is expressly intended notto apply to blank media, an unprogrammed processor or unprogrammedgeneric computer, or an unprogrammed programmable logic device,programmable gate array, or other unprogrammed device, unlessaccompanied by programmed media that confers the ability to theunprogrammed device to be configured to perform the disclosedfunction(s).

The foregoing description, for the purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the embodiments and its practical applications, to therebyenable others skilled in the art to best utilize the embodiments andvarious modifications as may be suited to the particular usecontemplated. Accordingly, the present embodiments are to be consideredas illustrative and not restrictive, and the invention is not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

What is claimed is:
 1. An apparatus, comprising: a memory; a processingdevice operatively coupled to the memory, the processing device to:receive, by an experimentation platform, an identification of a featureand an indication of a first event type associated with digital contentof a content provider, wherein the first event type identifies a firsttrackable action performed by a user with respect to one of a pluralityof variations of the digital content; responsive to a determination thatthe feature associated with the first event type does not exist in aconfiguration, generate the feature in the configuration based on theidentification of the feature; and log instances in which the firstevent type is received with respect to the feature of the digitalcontent.
 2. The apparatus of claim 1, wherein the identification of thefeature comprises a name and type of the feature.
 3. The apparatus ofclaim 1, wherein, to determine that the feature associated with thefirst event type does not exist in the configuration, the processingdevice is to determine that the feature does not exist in a cacheassociated with the configuration.
 4. The apparatus of claim 1, theprocessing device further to: generate a graphical user interface (GUI)for configuration of the feature; and responsive to a user interactionwith the GUI, configure an experiment associated with the feature thatprovides additional variations of the digital content that areassociated with the feature.
 5. The apparatus of claim 1, wherein theprocessing device is further to select a version of the feature toimplement based on the logging of the instances in which the first eventtype is received with respect to the feature of the digital content ofthe content provider.
 6. The apparatus of claim 1, wherein generatingthe feature in the configuration is further based on a comparison of anumber of features that have been generated to a threshold.
 7. Theapparatus of claim 1, wherein, to generate the feature in theconfiguration, the processing device is further to access arepresentational state transfer (REST) application program interface(API) to create a request to generate the feature on the experimentationplatform.
 8. A method comprising: receiving, by an experimentationplatform, an identification of a feature and an indication of a firstevent type associated with digital content of a content provider,wherein the first event type identifies a first trackable actionperformed by a user with respect to one of a plurality of variations ofthe digital content; responsive to a determination that the featureassociated with the first event type does not exist in a configuration,generating, by a processing device, the feature in the configurationbased on the identification of the feature; and logging instances inwhich the first event type is received with respect to the feature ofthe digital content.
 9. The method of claim 8, wherein theidentification of the feature comprises a name and type of the feature.10. The method of claim 8, wherein determining that the featureassociated with the first event type does not exist in the configurationcomprises that the feature does not exist in a cache associated with theconfiguration.
 11. The method of claim 8, further comprising: generatinga graphical user interface (GUI) for configuration of the feature; andresponsive to a user interaction with the GUI, configuring an experimentassociated with the feature that provides additional variations of thedigital content that are associated with the feature.
 12. The method ofclaim 8, further comprising selecting a version of the feature toimplement based on the logging of the instances in which the first eventtype is received with respect to the feature of the digital content ofthe content provider.
 13. The method of claim 8, wherein generating thefeature in the configuration is further based on a comparison of anumber of features that have been generated to a threshold.
 14. Themethod of claim 8, wherein generating the feature in the configurationcomprises accessing a representational state transfer (REST) applicationprogram interface (API) to create a request to generate the feature onthe experimentation platform.
 15. A non-transitory computer-readablestorage medium storing instructions that, when executed by a processingdevice, cause the processing device to: receive, by an experimentationplatform, an identification of a feature and an indication of a firstevent type associated with digital content of a content provider,wherein the first event type identifies a first trackable actionperformed by a user with respect to one of a plurality of variations ofthe digital content; responsive to a determination that the featureassociated with the first event type does not exist in a configuration,generate the feature in the configuration based on the identification ofthe feature; and log instances in which the first event type is receivedwith respect to the feature of the digital content.
 16. Thenon-transitory computer-readable storage medium of claim 15, wherein theidentification of the feature comprises a name and type of the feature.17. The non-transitory computer-readable storage medium of claim 15,wherein, to determine that the feature associated with the first eventtype does not exist in the configuration, the processing device is todetermine that the feature does not exist in a cache associated with theconfiguration.
 18. The non-transitory computer-readable storage mediumof claim 9, the processing device further to: generate a graphical userinterface (GUI) for configuration of the feature; and responsive to auser interaction with the GUI, configure an experiment associated withthe feature that provides additional variations of the digital contentthat are associated with the feature.
 19. The non-transitorycomputer-readable storage medium of claim 15, wherein the processingdevice is further to select a version of the feature to implement basedon the logging of the instances in which the first event type isreceived with respect to the feature of the digital content of thecontent provider.
 20. The non-transitory computer-readable storagemedium of claim 15, wherein generating the feature in the configurationis further based on a comparison of a number of features that have beengenerated to a threshold.